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• THE JUNIOR HIGH^aASSROOM ORGAtMZATION STUDY: 



Summary gf Traininp; Procadurgs and Methodology 



Abstract 



The Junior High Cla8|frooiji Organization Study /fas part of a col- 
laborative effoift tetweert Tfie Aixe-rsity of Texas Research -and Develop- 
ment Center for Teacher Eddcation, the Austin Independent School District 
(AISD), and the National Institute of Edvfcation. This, study wws designed 
to answer' questions abAut effective instructwri in junior high school 

• • • ^ 

EnRlish arid 'mathematics classes b)- fo^^using on classroocn organization and 
management, particularly those steps that areSgimportant at the beginning 
of the school year.^ Muth qf the methodoiogy^'and instrumentation^ for ' this 
study came from, a study done in third-grade classes in low SES scho^ols 
done^-hv^l977-78 (Evertaon, An^ierson, Emmer, & Clements, Note 1). During 
the school 'year 1978-79, data were collected to address these questions, 
and analyses of the data are" in progress. This report pretrents an "over- 
view of the history of the study tfnd briefly de-rcribes the data collej:- 
tion activities. More complete information on .the ^tudy can be found in 
Evert son, Emmer, and Clement8 (Note 2). / 
... ' Back^royndljP ^ 

Much vfork done at the R&D Center over the last few years by the 
<;orreUte8 of Effective Teaching Brogram (COET) has focused on classroom 

processes which related to achievement in the basic skill areas in 

* ',, 

»;lementary and iunior high schools* This interest and general research 

background led COET staff and p€r"gonnel at the AlSq Offices of Develu>v^ 

'■^ H • y if " 

mental Programs and Research and Evaluation to .ioiitlv tackle school 

■ • .fx ■ . y ^ ./ ■ 



problems of low SES ach ievemenL . Initial planning efftfiLs produced a 
series of research Ve views dn effective teaching ih lov/sES e^mentary * 
schools- (Other Center proRrama were al^o involved in otHeV octnponent a . ) . 
The reviews included research done in Aastin schools 'by the Center^ as " 

: • ' ■ "> ■ • ''^ '•■ \ 

well as work b^ r48earcher8 " in other parts of the (;puntry; They covered 

» • - 

the topics of in-service, c l-asardtwn, structure, teachinj^ tne^ods, teacher- 

^ii ' ■ . 

'student interact ion, and the use pf instructional time. Kagh waw pro- 
duced in twg, -forms:, a 15-20, page papfer, and a 5-6 page ^jfrsion that 
sunmarized ,the highlights of the research'. These reviews were 
disseminated in, the AI^D by the. Center an% by the AlSD.Office oS .Research 
and EvaiCation (ORE) with whom the- Center had worked , closely in preparing 
t^r^iews. Jhe papers we%e part of one of ORE' 8 priority efforts that 
year,: ^hich waa the* gathering of information ^or district personnel about 



, research *^indinR8 on instruction of low 5ES., student* . ^.-^ -prepared 

T • - .... 

summaries for o^her L<(pics, and the R&D Center prepared th<' X/^iews of 

'■ ^.-^ ■ \ ' 

research on claa8room''proces8es, 

" <»he result of tlYfesf Siummaries was that R&D Ofenter staff met with 

AISD administAtor^ in the Division of Instruction and began to discuss 

how' the CentJ^r researchers might be of further use to fJsact itioners in 

the District. -Vhese contacts resulted in a list of research .questions 

• t ^ 

about effective teachi^OR in elt.raentary 'schools, especially low SES 
schools, whi-ch wW of high concern to the staff of the Division of 
Instruction. 

One set o£ questions on this list was about classroom organizat ipn 
aind management. Since much past R&D research had yielded conclusions 
that these were extremely important facets of teaching, especially in low 

V 1 - ■ 

SES schools, it w^s decided that, this topic could be most effectively 



researched by pchraons at the Center. .Indeed, all of the rlsea'ifch 

summaries prepared by the Center,, and much original work done here,, had 

expressed the itt)portance df effective orxanir.at ion arid managetne^aL? . 

techniques which resulted in gi^tft-er student time in acade m/c taak» and 

greater involvement with and exposure to^academxc content 

However, very, little was known about* what specific teacher behaviors 

* . . V 
result in %etter" organization. In particular, little information was. 

available about what factors are most important in establishing a smooth-- 

running classroom, at the be'ginning of the^year. There is a wealth of 

general acjvice that ranges from "Don't smile until Christmas" or "Have 

your room organized and ready on 'the first day of school.'^ However, it 

felt Hat such general statements were not sufficient to help a 



vaa 



teacher learn effective organization, especially when s/he has never 
taught before, .General principles of instruction are .useful,' but they 
rau«t be illustrfft^ed and supported by c(3ticrete examples if they are t© be 
internalized by new and inexperienced jteachers^ 

i ] ( 

Therefore,' the Elementary So*\ool GUasroom Organizatior. Study was 

designed to.an&wer some very specific questTons" about establishing and . 

maintaining classroom organization that results in jj ^reater student time 

on task , exposure to content , and achievement . The ultimate purpose of 

the study was to -produce knowledge that coujld be expressed in terms of 

J 

specific teacher behaviors that produce effect'"" management of t_iine, 
instructional materials , contacts between the teacher and studf its, and 
the external constraints imposed on teachers.. These topics reflected 

8pecific district concerns. Tl)e study also yielded nev; ways of 

« ■» « < ' 

conceptualizing^ classroom organization, • \ ' 
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* After several informal discussiorta with AISD stnEf, a proposal was 

. ' . * 

sent to the Director of ElementaYy- Edocation which presented soine obiec- 
tivea which would be addressed by therstudy. The resppnse to this pnjn 
posal was very positive, ^nd it was ^Rreed to conduct a study during the 
school year 1977-78, to focus on or^nization and management in third- 
grade classes in schools with large propc^ftions of students from low SBS 
backgrounds. 

Principals in 10 Title I and near Title 1 school's were contacted and 
the proposed study vfas diicussed. with them. Generally,, the ^reaction of 
the principals was quite favorable, and they arrange^ <or a meeting with 
their third-grade teachers during the week before school.- After meeting' 
with faculties, 29 teachers in eight schools agreed to participate in the 

i 

Study, Twenty-seven teachers continued in the study until the^lfend of tHe 
acl;ool year* * 

r 

Analyses done on the third-grade data showed that management capa- 
bility during the first three weeks of school was a good predic(|or of 
management capability during the rest of .the year* It was found Xhat 
efl?ec:ti>fe managers differed significantly from less .effective mf^nagbrs in 
many beginning-^, and end-of-year management activities and behaviors. 
Data used reach these decisions were student on-task behaviors, reader 
ratir^ga based on carefu5 analysis of . nartative data , obpervers* end-qf- ^ 
year ratings, and class nean residual gain on the California Achievement 
tests* ' ^ • • 

BasivJ nn the positive findinxs from the Elementary School Classroom 
• Organizalion Study, the decision was made to study junior high school 



lasses- to 8H!C if similar restilts could be fpund : With the school 



district 'si cooperri *. ion, a similar study was set up to observe junior high 



'I 



school English and jiwkthemat ica teachera in all 11 of the -AuffCin iumor 
high schQoXif; extensively in the first three weeks (including the first 
day of school) and- approximately once every three weeks thereafter. (A 
time line of the atady is included in Appendix A.) The goal, again, was 
to ^answer questions about the. establishroenf and maintenance of effective 

classroivm management. ' 

Initial Contact* wuth District Administrators and TeAchers 
After receiving the approval and support of the Austin Independent 
School District's Director of Secondary Education and of the Office of 
Research "and Evaluation, initial contacts with nine of the U junior high 
school building principals were made in the spring qf 1978. The other 
two piincipals and schools were -contacted during late summet. During 
meetings with the building principals, the purpose and nature of the , 
'.Junior High Classroom Organisation Study was explained and the approval 
pf each building principal was received. The representative from the 
research proiecft then requested that the principal call a meeting of all 
English and math ^tics teachers so that the project /:ould be explained 
to thera and so that their participation could be so.ugKt. ' Subsequently, 
during Kay of I97fl, a representative from the project met wi^h each group 
of faculty in their building to discuss the project and t^ answer ques- 
■ tions regarding it. In each case, the teacher's participation was 

requested, and teachers were allowed to sign up for the study, which was 
to be -conducted commencing the following fall. Approximately three- 
•fourths of- the teachers indicated 3 willingness to participate, and a 
number of those who did not volunteeo indicated that it was because they 

« 

were not plahning to teach in thai building the following year. Thus, 
the available sample for selection of teachers was reasonably represen- 



tative of the total Rppulation.< Obviously, of course, there are some 
volunteer effects, althouRh they should not be too Rrcat, conaiderioR the 
hiRh acceptance rate* 

In order to include new teachers into the study, a representative 
from' the project arranged tovspeak to new teachers during the in-service 
days in the week prior to the beginhlng of the year. Most of these 
teachers were present at the meetings, a i approximately one-half indi- 
cated, (/illingness to participate, even though it was the^r entree into 

teaching. *' 

Final selection of the teachers from the available; pool alloijed for 
representation from each of the 11 schools in the district and roughly, 
proportional representation on.. the basis of years of tea^:hing experience. 
In addition, 17 of the teacher* who had volunteered had also participated 
in a large-scale ♦process-product research study three years prior to this 
study. All of the^e teachers Were included in the final sample in order 
to test hypotheses regarding stability of teaching behaviors and effects 
over time, as well as to study changes in various aspects of these 

'a 

teacher?' instructional characteristics. Within the constraints imposed 
by the preceding factors, random rfeledtii^^ was U3e4 to obtain the final 
sampl^t . > . » ^ 

Class ancJ time schedules were obtained from the schools for use in 
observer scheduling. After the selection ol teachers^ was made, & lett'isr 

n 

was sent to the' 53 chosen teachers informing them of the two periods 

selected to be observed, the names of the observers, ^nd the schedule of 

observers during the first week of school. Principals of the 11 schools 

* ' '\ 

Wre sent a fetter informine them of the' teachers and class periods that 

would be participating in the' study. Letters were also 'sent to teachers 



' ' '' 

not chosen m i}e participants thanking tfV-em for their willingness and 

■, » 
i.nt«re8t. The ta^chers and princi^^alB" in the study rieceived the schedule 

for the'second and thaid weeks durinj; the Utter part of the first week 

♦ 

* * • ■ , 

of school* ■ 

* 

Observer S e lection and Training . ^ 
Nineteen observers pi^rticipated' in the original' phase of data col- 
lection. Most of the observers weie svslected from a list of ^?raduate 
students in the Educational Psyohplogy and Curriculum and Instruction 
depastments at The University of Texas at Austin. One of the require- 
ments for selection was that they have classroom teaching experience. 
Some of the observers were formejK teachers recruited into the project,- 
some were observers in the t.hird-srade classroom organization sjudy, and 
others were R&D Center staff members. All observers underwent a week'i^ 
training which emphasized the nature of junior high clas-brooms an^ the 
types of teacher and student behaviors which were important to note^.* 

Training began a week before the beginning of school. Observers. met 
with R&D staff members for approximately 25 hours of training. During 
the training sessio.s the following topics were discussed: preliminary 
results from the Eib raentary School Classroom Organization Study, concepts 
•and terms u8ecK> in the study,, techniques for writing narratives, noting 
time intervals, use of the forms (Student Engagement Fating, Time Log, 
'and Component Rating), procedur'es for handing in materials, and how to be 
an unobtrusive observer. Each of the forms was explained in detail and 
then practiced, using videotaped observations of junior high or upper- ^ 
level elementary classrpoms.' These videotapes also were used for prac- 
lici! '.n writing narratives: The pract ici^. forms wert- handed in and 
checked by staff members, usually overniKht) and feedback was given to 



observers during the next session. Copies of desirable forms were 
provided ao that the (ibservers could compare their o.wn forms with what 
wasl expected of them. Observer practice forms were tfsed to check reli- 
ability also. Ample opportunity was given" for ^answering questions and 
open-ended discussions to be sure that observers felt comfortable with 
the forms and ^understood the rationale for using them. ^ " 

Among the, materials ptoVided the observers was a notebook for' use 
duriHR the observations (Evertson, Emmer, & Clement|, Note 3). It 
contained all the nyterials observers would need for conducting and 
completing their' observations. The notebook was divided into five parts: 
narratives, time logs, student engagement ratingfl, component ratings, and 
rai8cey.aoeous. Each of the, first fouf sections included a set of 
guidelines, a sample filled- in form and blank copies of the fofms. 
Observers were instructed that additional copies of each form could be 
obtained as needed. , taAi form had a s^ar^dardized ID field. 

The miscellaneous se^fction incj."ded a map of the city with the loca- 
tions of the junior ^ligh schools marked and instructions on how, to get to 
each one; general guidelines for the observer^; a list of. standard terras 
used in the study; a list of teachers, observers' and -code numbers.; and 
other bookkeeping items' such as mileage forms. Observers were also pro- 
(rid'ed a tape-recorder and blank tapes for recording tlie narratives.^ At 
the end' of the training sessions, obselrvers wyre given their schedulea 
' for the following weeks. , 

Class^dom Observations in the First Three Weeks 

. 1 ' — -~~ — ' — ' ' ~~" ~~ ' 

'.Each of the ori^ginal 52 participating teachers was seen abt»ut 10 or 
11 Limfs rfiiVtng the first three weeks, of school yielding-over 575 obser- 
va»:ion hours. One class of each teacher was seen on the first, second. 



and fourth days 6f the first Wcek of schools , Another class was observed 
xjn the, third and* fifth' ilays of that week. Thereafter, each of the two 

.c'laases was seen about eiqually. Each teacher w.-'.s seen by two different 

• '* . ' \y ' ' ' ■ 

observers. 

• ■ >" ' • , • ' ' 

The obBervation^ sessions in the first three weeks resulted in four 

types of information. ..Samples of the forms are included m Appendix B of 

this fepoftt _ • , 



) ■ ■ I. THE NARRATIVE RECOftD 



While in the classroom, the observers maintained a written record of 

■ I , 

classrooitt events, with a particular focus on organization and management, 
• - • ' *♦ , 

The tr?'"*'*s manual, including , guideline questions, was used for 
xef«j»jttC«.%urpo''e4 while the observer was recording his/her eSmments in 
greatet detail , following the observation. The- narrative technique, used 
in the 1^ird'^iSpi;*'Cla88room Organization Stud^ allows info rtnat ion to be 
gatti\ered «bout a wid^^rr-ay of classroom events. The math areas covered 
in j;))e narratives or in comments following the narratives were: leader- 
ihip in controlli1% classroom behavior , instructional leadership, 
instructional systems in operation, student concerns, physical arrange- 
ment of Che room, constraints on the teacher, personal characteristics of 
the teacher, visible students, and peer interactions. 

Following the observation, each observer recorded the narrative on 
'tape and made comments at the'end, if necessary. It was found that by 
doing this, far more detail could be recalled , than could be written 
during the oroservation thus providing irore detailed information. Narra- 
tiv«f,d€ «tich observer were read during data collection by R&D Center 
staff roerabers in order to maintain a high level of quality m the 

» 

description*. Observers were theitvgjven feedback on areas whLch| needed 



more itteation. A typical specimen^ record lenj^Ch js eight to ta^n type- 
written pages for a 1-hour obsei^ation. ^] », 
■ 'II. STUDENT ENGAGEMENT RATINGS 
\ Since short-ti^rm j^utcoraes of student t irae-on-task were important to 

'i , \ r- 

thid study, the obseryers filled out a set of ratings which classified 
students in each of eight categories^ of engagement. The first count was 
taken at a randomly selected time within the first l^ minutes of the 

t • 

class period and each 15 minuted, thereafter,. ^Students were classified as 
follows: 

/»a. D(^finitely on-tasW academic . The student was working on an 

■ . . ' /. . ' ' 

academic^ assignment or receiving an academic presentation and very 

% . ■ - 

cleaj^ly paying at tent ion • 

/ . ' , * 

b. Probably on-task academi c. The student was supposed to be work- 

ing on an academjlc assignment or**^ tending to an academic presentation 

but could not confidently be said to be jpttending; however, he was not 

definitely off-'task either (e.g., staring into space as if th joking ahput 

• r • 

the task). 

c. Definitely on-task procedural . The student was performing a 

# 

procedure or routine which wasV preparatory to beginning an academic 
activity, oV was necessary for finishing it (e.g., passing in papei^s f^r 
a transition), 

d. P robably on-task procedural . The student was probably engaged 
in some procedural activity, but was not clearly doing- so; however, was 
not definitely of f-task ^either' (e.g., moving across the room, probably to 
pick up matiprials). . 
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e. Off-^task sanctioned v The stucjent was not perfo' ming an academic 
or procedural tasHi but was not misbehaving (e , Roing to the wasfr^r 
basket). • ' ^ ,1' ' - \ - • 

* f'. Off-tagk unsanctioned' . The student was involved in an unde- 
sirable activity or\io1^ involved in a desirable activity (e . g. , Atalkingv 
when this was not allowed pt olfearly not doing an assignment). 

g. S ead time . . The student did not have anything speciific lie was 
supposed to be* doing (e.g., ^j^aitinrg for* the rest of the studeats to 
finish taking a test). * * > • ' ' 



h. Can^t tell.^ When, the observer could not confidently classify a 



,«tudent a^'^'^lpngi^ng to one of the above categories or when the student 

could ot be seen, he or sh^.was c<jsunted here. . ^ - ' " 

y ' ^ ■ ^ 

These JEorms Were alsd coded to reflect the £prr|^t of/ t;he activity 

in the classroom (e.g--, teacher- presentation to class, academic,, or 

indiyidual activities), who was in charge (e.g., t^achjer in charge, other 

adult present, or student teacher in charge, teacher present), and what"^ 

.the topic was. I *^ * ' T 

' ■ - ' s > 

• III. COMPONENT RATINGS f , 
- ^Vlthough the narrative records" wer^the richest source of data ^ 
regarding the teachers* organijcational and management behaviors, a sys- 
teraatic set of counts and ratings of specific characteristics was com- ' 
pleted^fter each observation in or<(er to provide a commo^ set of mea- 



sures for each teacher. JTherefore, after each observation, a set of 44 
^ratings was completed. (During the first and second week of school, only 
36 variables were rated. It^ was felt that additional information was 
ne^^ed regarding the teacher S management of student behavior, thus an 
additional eight ratings were added after the se^Qv*. week.) 



^ . Th^s assessment system had been used in the Elementary Schcwl Class- 
toom Organization Study as a raea^s of identifying global areas that d^f 

r 

ferentiafec various levels of capability irt orj^anizing and structuring 

♦ 

classroom activities. Also, the use of'tl ? assessment system provided a 

way to relate this research study to preservice teacher education, since 

an earlier version of the assessment system had been developed by a 
,1 ' ^ * » . 

committee of faculty from The University of Texas at Austin, including 

(ine member of the AISD Pri^ject staff. ' The system was used on a pilot 

basis in the elementary teacher education program at The University of 

Texas; research on this ^system will* allow^ these preservice ctite^ia to be 

f . • 

validated against teachers *in the jEi^eld. Seyei^al 'altet'at|ons and addi-" 

>, ^ 

t ions were. made to the original torm to reflect the different age levels, ; 
being observed and the different types of information desired. The areas 
focused on were: lesson design; lcfc#ting, constructing, using materials; 

• m ' ' ' . . . 

presenting information; developing attitudes; managing pupil behavior ;^ 
Methods cal^handling^ disrup&jive behavior ; interact ing effectively; class-^ 
roor? climate; aijiount of inappro^piate behavior; and teacjier'i reaction to 
inappropriate behavior. • * 

f , IV. TIME L0G6 

Time use was another important aspect of this study. It was. felt , ^ 
that an organized teacher would have maximum time on academic instruction 
and very little tute spent in transitions between activities an^ 
different fiubject matt^, and little or no dcdd time, that is, tim<*-in 
which students had no assi^ed activity. On the narrative form, there 
was space to the left of the numbered lines to be used to nyte tiroes such 
as the beginning and end of transitions, dead time, and. interruptions. 



Narrative . / 

Teacher # XX NSchool #J<X Sub\iect 1^ XX Period XX Obsefver # XX 
Date 8/30/79 # of Students Present 19 ' " Pa^e _J_ of _8 
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BEG 
T 

8:05 


END 


ST. ENG 

» 

> 


— — ' ^ — ' , 

t 

1 . the students have^ come in^ early and are 








2. fitting in their seats talking quiatly. 


_ . . . , 







3/ (Bell at 8:05) Some students talk to 




T 

8;06 




A. teacher; th jse others start working ands / 

■ ^ , , — , ^ - 








5/ reading iwit'i aide. Students know what to 

^ ; V^— ; 








6. flo already and get busy working at their 
1 ^ — , — , , — _ . — . 






V 


7. 8ea,t^ on reading activities* 



These tirnesrwere inserted into the taped narrative to. provide a feel 
for the flow of events in the classroom when the narratives are re*id. In 

. - V ■ 

addition, k time log for each^ observ^dtlcfn was completed by the ohservei;. 
On thi» form the observer noted the start time, stop time, number of stu- •» 
dents invalved, and a brief description of each activity in the classrDora 
(inclucfing transitions and dead time). The beginning titne^on the timS 
log was the bell beginning class and the end time was the bell ending the 
. class, with occasioi^Uy some notes conceri^ng hoy time was spent before 
\ and after class if it was relevant. This time *log einables the readei^j^ho 
follow the schedule throughout the class period for all of the students 
in the class. From this time log, one can see how much time ^wAs, spent on 
academic instruction, versus -time in transitions or other non-academic 



0, 

'<itt ivities. 
Summary 



To summarize, by the pnd of the first three weeks, each teacher had 
been observed at least nine times in two of his/her 50 to 60-minute 
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classes, and each observation resuUecf ih' a detailed narrative, student 
enRanement r>^njiv3, component ratings, and a time log. These data repre- 
sent an intes^e examal^ition'of. the beginning of school in English and 
math classes in Junior High School. , 

O^her Data Collected fr6m Observers After the First Three Weeks 
As a result of the observation- schedule, each observer saw six 
teachers about five vtimes each, and each teacher was seen by at least two 
obd<?rver8. In-order to get additional impressions and information, each 
observer provided several suramary ratings and descriptions of each 
teacher s/hft observed. The observers completed a Summary Component 
Ra^inR form (using essentially the same form used for eac^i observation) 
ip which they assessed the teachersydassraoms according to their overall 
impr^sion. The rating was based^ on a scale of 1 to 5 (1 ■ not at all 
characteristic of the teacher, 3 « moderately characteristic of the 
teacher, and 5 - highly characteristic of the teacher). In addition, 
'Observers answered six questions dealing with* general impressions about 
the organization of the classroom on each of his/her teachers anti, two 
general questions' regarding advice to new teachers. F\nalV observers 
provided a copy of each teacher's rules, a drawing of the room arrange- 
ment and I copy of the two observed classes' sealing charts. These items 
served as an aid to the observers who saw the tea^heVs froin Week 5 
through the end of the school year. "^ 

Classroom Observations A^ter^ the First Three Weeks ' ^ • 
After the first three weeks of school, observations stoppled for one 
week. During this C^me.'j^'^ns were made for the scheduling of .observa- 
tions for the rest of , tt>e school year. Fivi; observers fi;om the original 

r 

'/group of 19, in-luding the s^aff member In charge of tramift^^and managr 



ing the' stxidy, were chosen to' continue observing for the re^t of the ^ 
yqar. Observers were ohoaenvwho possessed good narrative-writ ing skills 

• r • ' ^ ^ ' f ^ 

and good teacher rapport. 6f the original 53 te^AcberYin the study, 51 
cont'inued to participate during the rest of the year. The other two 
teachers transferred out of the 'district . I A new system of schedujtng was 
developed ms wel\ as a systejn for contacting teachers about the schedul-^ 
ing. Using the .-choj?!. district calendar, the remainder of the schopl 

year was divided Intp e.iRht|;^roe-weekVcyc les • Short school weeks, due ^ 

• » ^ " 

to hdlidays or teacher cyrr^culum .days , were used for roake-^up observa*- 

tioni. Each of the^five observers was assigfied eight to 11 teachers , to 
observe, and both clasVperiods of each* teacher were scheduled* to be seen 
once 6very cycle, or approximately once every three weeks. Tea.chers were 
sent a list of scheduled' observations tor their school during the last 
week of the previous cycle. Basically, the te^icJjier was seen dul/lng the 
same week of each cycle (i.e., first week of the cycle, etc.) although 
the day of the week would ^^oifferentA A rotating schedule was set up 
to incori>orate observers' scheduling demands, but an attempt was made to 

see all teachers on a variety of ^.days of the week. Any changes in the 

. f r ; 

^scheduling were made through the office, and teachers were contacted when 

observers coul^d not make it to their classes* For the most part, make-up 

observations wer^e rescheduled as soon as poss^lble. Teachers were » 

... . - ■ ■ K 

requested to inform the office of a>y changes in .school s(;hedules so that 
observers could be notified ancl^^heduling^done. After the Christmas 
liul^ys ^he schedule, was changed Boraft&hat \iue to increased work time of 

om? observer . , The. observations * from the last weeK of ♦Sf>» tember to the 

• ■ * ♦» ■ ' . 

end of schbol resulted in an additional 850 hours of observations, or 

approximately 17 one-hour observations per teacher. ^ 



^5 



i. 

The observations were resuraed after the week- long break. The pur- ^ 
pose of following each class through the year was to assess continuity or 
changes in organization, methods of instruction, teacher and pupil behav- 
ior-, and short-term outcomes. No major changes were inade in the forms or 
techniques used; therefore, each observation continued to consist of a 
narrative record, a 8"et of student engagement ratings, a set of component 
ratit\gs, and a time log. In addition, a system for checking observer 
reliability was established. 

Reliability , • • ' 

During the three 3-weSk cycles ^rior to Christmas, eacfi observer saw 
two teachers normally seen by another observer.^ A regular obs^vation 
was done and all. forms filled Jout. The component ratings and atudent 
^eng4;emcnt ratings were then checked against thte summary component rat- • 
infes and a sampling of student engagement ratings from the first three 
weeks. It was expected that student engagement ratings would not aig- 
nificantly vary from ^he first three weeks. Allowances were made for 
differences in activities in checking these ratings. On the component 
ratings, a check was made to sCe which dimensions had a variation of more 
than one point (on the scale of 1 to, 5) from t)ve summary ratings from the- 
. first three weeks, if the two previous observers agreed. These dimen- 
sions were tjjen checked against the written observation to see if they 
were consistent with what was written. These, dimensions were also dis- 
cussed with the 'observer and attempts made to be sure that all observers 
tjnderstood. the ratings and were rating them Jons istent ly . Reliability 
and scheduling were discussed with observers at regular meetings, held 
once every three weeks and called nw|t» often if needed. 
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Afttjr Christmaa a new system was instituted. During each cycle, 

each Qbserver saw a teachfe. with the regularly "Bcheduled observer. 

Arrangements were rtiade ' with -the teachers in advance so 'that the observers 

could sit besid'a each other and have the same vantage point for taking 

' student engagement ratingti. The observers w^i\ld agree at wiiat time to <\o 

the ratings but they would not consult with ea^zh other while doing them. 

In addition to the narrative and student engagement ratings, observers 

also did the component ratings and time logs. The student engagement 

ratlng8~*Vtar©vthen compared to see if there was observer agreement. The 

component ratings were checked against each other and differences of more^ 

than one number'^w^e discussed'; with tHe observers and checked aga-inst the 

\ . \ . 

narratives. Changes- W)^re\ not' made on any of these forms. Discussions 

• .1 , ' ■ • . ^ . ,. 

were held in the regular ly scheduled .observer meetings, to maintain 

reliability. Twenty-thr^ aych checksvwere conducted. 

Spot checks were also done^on the narratives to be aure thaf 
observers were providing- the desired detail and" cj^vfe rage "of all major 
points. ' . ■ 

Co ntacts with Teachefs ' 

Additional contacts were made with the teachers during the remainder 
of the school year. Teachers were informed of the reasons for and 
scheduling of reliability obser/ations and other bits of information vira 
notes attached to observation schedules. In addition, a letter regard- 
ing achievement testing to be done. and other end-of-school data coUec- 
tion activities was mailed to the teachers in February. Observers 
directly contacted teachers to set up Convenient times for the achiQve-. 
mcnt testing and interviews, i 



Othdr Data Collected 

St udent Rating of Teacher 

Durin«5 t\\e. last regularly schteduled observation o£ each class,, 
observers administered a student rating fcrrm. The form was adapted from" 
the Student Rating Scale o£ Instructors developed in 1,973 by the Sequoia, 
California- High School District (Stallings, Needels, & Stdyrook, Note 4). 
The procedure took approximately 10 minutes of class time atffi the teacher 
waB not present while the rating was given. Stu(jents were told that 
their answers would be kept confidential, that their teachers would 
receive only clflss percentages for each question after the end of the • 
school year. The observer read aloud the 17 questions on the form and 
all of the five alternative ans.wers. Students wer^ instructed to choose 
the answer which best described how tiky felt about that class and that 
tbac^er.' Questions on the form de^lt with instructional' and behavioral 
organization, as well as teacher style. Students, in general, were very 
cooperative and, serious while doii>g this. 
Achievement Teste 

- Achievement • tests vfere developed to measure student learning and in 
a manner to assure both ccMttenf validity and adequate reliability. The 
construct ion -<^f the mathematics achievement test^was based upon curricu- 
lum materialp used in seventh-^nd-eiRhth wade math classrooms in t:he 
school district.' The district-wide adopted textbook series for junior, 
high math wad /Examined and mult iple-choice 'items were developed to 
reflect the j^reas of emphasis in these texts. Hems on the preliminary 
test form were submitted to the Coordinator of Mathematics Instruction 
for the school district^ and fe\jdback regarding the appropriateness of 
these items was incorporated into the test development activity. The 
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mathematics achievement test was divided into two subtests each of 20 • 
minutes duration* The first covered mathematics compivtation including* 
whole numbers, fractions, ^decim^ls, and percent^. The second subtest 
covered mathematics concepts, applications and reasoning with items drawn 
from the areas of geometry, number theory, probability and statistics, 

r 

verbal problems, integers, and other areas ^covered in the curriculum. 
Th<i. final test contained 80 items, with a number of both easy and^diffi-* 
cult items in order to avoid ceiling and basement effects. 

The English a'chie\^eraent test was constructed in a similar fashion. 
A preliminary form of the test was submitted to the District Coordinator 
for Lai guaf?e Arts Instruction and connnents. regarding .the adequacy of item 
g^ampling were incorporates:^ into the final form of the test. District-- 
wide adopted textbooks for English instruction in seventh and eighth 
grades were consulted during the development of test items. An attempt 
was .made to provide adequate representation of all areas of junior high 
English curriculum with the exception of that portion of the curriculum 
dealing with the interpretation of poetry* and other forms of literature. 

The English test assessed other areas of primary emphasis in the 

f 

^istrict-^ide curriculum. Tnese involved the various areas of gratwaar, 
usage, and mechanics, including parts of speech, punctuation,, capitaliza- 
tion, etc. In addition, several questions related to library refrence 
use were included, since this is generally covered in the juriior high 
curriculum. T|ie final test consisted of four parts to be administered^ 
during a total testing time of '0 minutes. Items were cho8fe;rt to'repre*" 
sent all levels of difficulty. Thus, for example, the spelling test 
included a ^number of items at the third-grade difficulty level on up 
thtimgh har-d-to-spel 1 words from the eighth--grade text. 




Both the English and the roathematics achievement test^ were pilot 
tested in, classrooms which were not partioipatinj? in the regular observa- 
tion study. The pilot testing indicated that the tests had-good internal 

i 

consistency reliability, both in 'the total score and in the subscale 
scored. Some items were modified as a result of the pilot testings and 
the mathemflticB test as shortened, somewhat when it proVed to be too 
lorij;. The pilot test indicated ^hat the. English achievement test could 
be lengthened somewhat, so about 255: mere, items were acided to each 
subtest. . ' 

The achievement test was ' Administered* during the two weeks following 
the last cycle of observations (the first two weeks May). The 
observers gave the test to the classes they observed. In most cases^ the 
teacher remained in the classroom during the achievement testing. The 
students recorded their answers to- the test items on a mark-sense scoring 
sheet. Students were proyided a test booklet, an answer sheet, a Numb-ar 
2 pencil, and a piece of scratch paper (math* only) during testing. No 

I 

incidents wei6 reported by observer-administrators that suggested that 
the conditions during testing would yield invalid information* High 

9 

absence *ates were reported in a few instances, so that follow-dp testing 

♦ 

was necfessary. A test administrator visited six of the classrooms 
app^roximately one week after the original testing and tested ihose stu- 
dents present at that time who had been absent during the first test 
administration. 

The achievement test data was used as one of the product criteria, 

A • 

after adjustment for initial achievement levels as measured by the CAT 
results from j:he preceding year. 
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other Data Collected from Obs nrvers at the End oj^chool , 
DurinR the period from the fifth week of school to,, the end of 
school, each teacher was seen approximately 17 times. All but four , 
'teachers were seen at least once for a reliability observation. ^ ^ 
finiwnar y 'Com|>onent Rating - ' 

On this 'form the observers' made a final assessment f the teachers/ . 
classrooms alcordinj. to iheir overall .impressions just as was done after . 
the first three weeks of school." A^ain. the rating was based op a scVjle 
of 1 to 5 (I - not at'all char^ct^rist^c of the teacher and 5 highly 
characteristic of the teacher). These ^itings were compared to the ^ ^ 
Summary, Component Ratings frocn the first three week's, as well as 'to ^the 
ratings from throughout the yea'r , 

, 9 

Obflerver RatinRg of teacher 

In order to gather further information. on the teachers not av;iUble 
through the,Component RatWtTTmc^es, etc'., the Observer Ratings of. 
Teacher was created. This was a compilation ;of questiuna. many of which 
were selec'teji from previous studies conducted by COET. W questiotis, 
dealt with classroou arr^gement and atmosphere, m^na^ffement; techniques,. 

teacher-student interactions, instructional techniques,, use of time. 
Tl^ese served to give a genfe^al picture of the teather.. The form- con^ 
* Bisted of '303 questions, and each observer filled out one on ea* of the 

regularly observed teachers . A^ a rough check on observar agreement. . 

son« observers from the first three 'weeks completed the ORT on teachers 

they had seen frequently enough to form a U ^np. impress.i,on. 

Teacher Competency Checklist 
' ■ The Evaluation of Teaching Program^ <EOTy wa? also involved ;n a 

pruieet with the Austin Independ\.nt School District to systematically 



evaluate the new disHrict-wide evaluation system put into effect this 
year. % form and a aeries of behavioral descriptors were- developed to 

I 

ev.aluate new teachers and other teachers up for evaluation on a three- 
year basis. As a part of the Collaborative School-Based ^I'oiect, our 
dbsfe^vars were asked to fill out a shortened virsion of this checklist 
for the purpo^se of comparison to certain teachers' evaluations don- by 
. th^ir principals. Observers wtre to rate their regulat^ly observed 
teacher^ cm a 8ca4e of Uto 5 in the babic areas of personal qualities, 
instructional skills, sclassroom management skills, expertise in basic 
skills and subiect area, and interpersonal skills (a total of 52 vari-' 
ables). The behWioral descriptors used by principals to filltout the 
forms weVe,also used by the observers' to fill <Hst the ratings. Based on 
performance information, observers were to estimate the teacher ' s 'effec- 
tiveness in meeting each of the crij:eria.> The rating scale was designed 
HB follows: 5 « a superior performance level ^expected , to include only, 
about 5% ol the professionals); 3.- the normally expected level pf per- 
formance (including approximately 50-70% of the professionals); and I - 
an unsatisfactyry-^performance level, t)ne whi^h must be improved (expected 
Ro include only about 5Z of the prof eas ionals ) . In addition observers - 
were to rate on a e_cale of I Xo 3 as to how confident they felt m rating 
the teacher on each particular ite/n.^ based on'^su/f icient inf<^rraation (3 - 
very confident in making the rating; 1 -notfvery confident An making the 
rating, insufficient information /to be a.ure). 

' • L • ■ ■ 

Teacher Ir^terviews and Quest lonrtairc ^ 

• While the observations are a'rich source of information, there was 
much ii^fotmation that could not be obtained through observations, 
especiaUy about planning, ' dec i«8ion-maK mg-, ^nd cons traints i«f feet ing th , 



teacher.. Therefore, each teacher comVtet.ed extensive questionnaire 
and was intervieweH at the end of the school year.; 

Interviews . In the middle of October, three math hn\ three En?,lish 
teachers, identified ^ effective managers, were chosen , to be interviewed 
for the purpcaii of obtaining information on beginning-of -school planning ^ 
and activities. In addition, each teacher was ^sked to describe the f 
students in ^J[t^ two classes we were observing and make predictions for 
their academit achievement. It was hoped that behavioral problems 
described by the teachers would show up the narratives, and specitijc 
ways of handling these problems would be observed and documented. 

During the last month of the school year, all §1 teachers were 
interviewed to obtain information on planning for^ihe begmning of 
school, "goals and planning for the rest of - the- year ,. relat,ive su^^cess 
rates for the observed classes, contacts with other school personnel, and 
the teacher's reflections on the school year. In addition, teachers weire 
asked to describe memorable behavioral problems and what was done to deal 
with them. When a systematic reading of the narratives is done, these 
sKidents will be picked out (if, indeed, 'they do show up) an-i the 
teacher's methods of dealing with them will be analyzed. In general., 
these interviews provided information about the teacher's e. 'ectations 

•for and assessment of their organizational systems, instructional plan- 

'''' 

ning, and specific students^ All interviews were done at the teachers 
convenience, usually in their classrooms, either during their off-period 
or after school. Observers interviewed the teachers they had regularly 
observed. All interviews were tape-recorded and later t anscribed 
vt'rbatim. These inturviewK will be Analyzed and content-cuded at a lator 
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It should be noted that observers were trained for the intervi^evfl^ in 
a session with COET staff members and with a member of the CBAM Project 
staff whose specialty is conduct ing interviews. Ideas for putting the 
teachers at ease,, conducting the int»:rview and giving some feedback to 
the tefichers were discussed. It was felt that the observers would be 
best able to conduct the interviews with the teachers as a result of 
their year-long relationship. Observers also were more familiar with the 
functioning of the classrooms and were ab'le^ to piobe for specific answers 
with the context of the classroom in mind. Observers were instructed to 
give feedback only when they were able to make accurate positive state- 
<men,ts and teachers were told that they would be receiving more informa- 
tion at a later time. ^ 

Quest jonnaire . A four-part questionnaire was given to the teachers 
to be filled out at the end of the school year. Questions in the first 
two parts dealt with items such as materials, grading;, Instructional 
^emphases and techniques, teacher-student rel-at iop^ipsj etc. On most of 
these questions ceachers were asked if they used 'these materials or ^ 
techniques and bow frequently. 

The third part of the questi^onnaire was a Teacher Concerns Checklist 
developed by the PAEI/ AM staff at the R&D Center (George, Note 5), . 
' This checklist pt;ovided information about teachers' concerns about >their 
role as a teacher • 

Tha fourth part gf the queVt lonnaire was an Educational Opinion^ 
, Survey based on a quest io'nnaire develop(jd by Wehling and Chatters (1969) 
on tt»acher beliefs about the teachin?; process, \ 

Teachers were also asked to fill out a 1? lograph ical information 
sheet aiid describe the effecr of the observer^on Lheir classe . 



Feedback to.Teichers 



In early August, foUowinR the sAidy/the 5^1 participating teachers 
were sent a packet of feedback materia\^., The packet consisted^F a 



letter thanking the teachers for participating in the study,, the percent-- 
ages of answers to thS items on the Student Rat ing of Teachers done by 
their two classes, the achievement scores for .their two classes and study 
means, summary information on t;he teacher questionnaire, a list of con- 
cepts and tferms u^pd in' the study, and preliminary results'* from tfie 



Study, *^ 



^ Pflta Analyses "^vV 
This section summariaes data analysis procedures already completed,, 

in progress, or proposed. Where results >of data analyses are known, they 

are briefly described and summarized, ffore extensive dis<:ussion of th6 . 

results, alo^g witji appropriate tables, will be available in separate" 

report form in the COET report series, 

iiie data gathered in the Juni^^ High Classroom Organization Study 

consist, of the following: 

1, Written narrative records (specimen records) of two classes for..^ 
each of 25 English and ^6 mathematics teachers ., teachers were 

^ observed-i^n one class on the first May of l^hool, ^nd on a total of 
approximately 10--12 occasions during the first three weeks. For the^ 
remainder of the year, each teacher was observed ^very^three to four. ^ 
WL^»ks on two occasions, once -in each class. 

2. Component ratings . A set of ^ , scales was used after each 

observation to raL'e a variety of lastrujt i'bnal and managerial behaviors. 

^ f 



3. Student hngagcment Ratings . Th^s6 assess- time-'on-task during 




each observat ion. 




25 
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4* Time logs . Records of the use of time in various activities and 
ftroupings* 

5. Achievement and atfitude, measurgs * Specially constri/cted 
achievement tests were administered in each content area in May. ^Student 
Ratings of thci Jeacher (SRT) wer*e also administered^ifi May in each class. 
Entering achi^^yement scores were estimated by'* the California Achievement . 
Test (CAff), adftiirtistered/the previous spring and made avai^labie to the 
project by the school district. j - ' ^ 

6. Summary ratings and .checklists. At the fend of the year, 

_ — — ) — . . ' : 

observers rated or classified teachers on a variety of variables. 

?• Teacher quest maire . ^ach te»Qher^ Completed an extensive 
^ . questionnaire consisting of measures of ^^ttit^^udes, beliefs, concerns, aAd 

1 • • ■ ■ ' ^ 

perceptions. 

8. Teacher interviews . Each teacher^was interviewed »in May, and a 
verbatim transcript' of the'' interview was produced. The irtterview focused 
on organization, .management , planning, and related ax^a^. 
Preliminary Analyses . <v 

A series of analyses w^re conducted to deLermine the reliability of 
many of the observational measures. In addition, several the variable 
KPts w(»re factored in order 'ta determine dimensionality and to select 
variables for grouping to iotva> acales. These analyses were undertak^^n 
preparatory to analyses addressing substantive questions, in order to 
ascertain the quality of the d6ta, and to reduce the data sets to 
manageable sizes. ^^ • . 

•Between-observer .agreement was verified by comparing Component 
Ratinp^s and Student Engagement lutings of observer pairs during twenty 
three reliability checks during the year. Usinj? the intra-class corre-^'N, 
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lation statistic to estimate observer aRreement /moderate to high values 
were obtained for nearly 0,11 scales /see "Tables 1 and 2). Within-teacher 
stability was estimated using measures obtained in di f ferenri per iods Cor 
-the same teacher. Most of the Component Ratings and Student Engagcmcr* 
Ratings' fexhibited at least moderate stability (see. Table 3). 

4 The reliability, of the achievement and attitude measures was deter- 
mined using the coefficient measure of internal consistency. Both the 
pilot testing and the study data indicated hjgh reliability of these 
measut.es. 

Residual achievement scoras wefe calculated for each class, using 
the CAT class rae^ as the coVari^te^^^Aftgr partialing out the entering 
achievement ^levels, the residual achievement gain showed significant 

< 

stability within teachers (between periods), .indicating coneist^ncy in 
teacher effects from class to class.' Student attitudes also exhibited 
consistency from class .to class, within teaci^ers, indicating thai the 
Student ratings of a given teacher were stable <|rom one period to 
another. The results were similar in math and English classes and are 
rep|)rted in Table 4. Correlations between the SRT and achievement 
scores, and between SRT and residual achievement wefe negligible. Tho 
intercorrelotion among the achievement test, the SRT, CAT, and residual 
achievement, measures of on-task behavior and inappropriate and disrup- 
Live student behavior are listed in .Tab^e 5, ' These intercorrelat ions 
vjere computed for both math and English .for the beginning, as well as th 

i ...... 

ond, of the school vea^r. ' 

At least two patterns' eraerftdd. AIL m^naReroent variables' vtire 
** 

significantly related to residual achievement in math after the first 
three weeks of school. This pattern did not appear in English cLasses, 



although the correlations were in the same \ 'edicted direction. In 
JSnRlish classes /\lnanagement variables ^ere signif icant^y related to the ^ 
student attitude^^measures . These relat'ioriships appeared for the first 
three weeks of school, as well as the rest of the year . in English 
cla^e's^ 

Analysis of Teacher Quest igfida ire 



The part of the teacher questionnaire d6ta assessing bcl^ijjjefd. and 



attitiides was -factor analyzed >and rocTD d to f't an a prion sJrtuctu<e, ^ 

A poor fit ^jAi^^ed, /so afi* exploratory factor analysis was performed* 

Ultimately, four scales were, created to summarize this set of dat^. 

The narrative records were beinj? summarized according to an activity 
. , • ^' 

analysis bashed upon Doyle's work. In addition/- several narrative summary 

ratings were developed fqr use by readers to condense information present 

in sets of narratives* ^ . , 

Seleclii n of a Subset ot Moi?e and LesB Effectlrt'e Manager a 

I ~T IT 

sAection of subsets of more and less effective . math and English 
teachers^ was made using multiple criteria. The criteria included 
adiusted oupil achievement means; SRT mdans; a management score derived 
from an obsenyer^ und-i»>f-year assessment; average percei.^age of adjusted 
pupijl achi^vfertfe'M me.ans; SRT means; a mrfflagement score derived from an 
obset^r end-d^-year 'Assessment ; avjrage percentage of unsanctioned, off- 
task behavror; and the percentage of time in -kcademic tasks during the 



October ^*>^>|ay observation period,. 

In order to avoid selectioji bias (e.g., confounding lyf^^tial 

ability or achievement levels with designation as a more or less effec- 

. ? ^ r . ^ 

tive manager), classes were grouped according to entering CAT means, and 

subsamples of more aitd less effectivi^ j^ianagers were selected within high, 

r 7 » 



middle, and 1%^ initial CAT levels^. In English, seven noce ef fective^.Nand 
seven less effective manaRers were identified^ in mathematics, six more 
ef fectiv^ 'anii sixrleas effective managers were identified. 
Datal-Ai^'alyses Directed at S ibstantive Questions J 
/ 1 . Relationships amoiR ttie various sets of , variables . A set of 
re'l^ed questions i-s vJhether" and to what extent the various sets of vari- 
ables are related. For example, are the teacher behavior variables, such 
as the Coiponent. Ratings, correlated with student process or product 
variables,, such as the engagement rates or residual achievement? Data 
analyses undertaken to answer the^e and other related questions used 
correlation and multiple regression methodology. Other analyses which 
compare different times of . the year, and different sets of variables are 
also being undertaken, ' . 

2. Identification of beginning-of-year dimensions of effective 
classroom management . Several analyses were directed at thi^ concern. 

, The subsaraplea of more and less effective managers were compared using j 
data collected during the first three weeks of the year, including 
narrative records, component ratings, and student engagement rate'5. ♦ 
Numerical data were analyzed via ^-tests and ANOVA, with case studies 
baaed upon narratives used lo illus-trate b^isic principles. 

3. Effects of entering student achievement level on teacher 
behaviors, activities, and management strategies . It i/j well-knuwn that 

A low-ability classes are perceived as more difficult to teach. Their 
fects on the teacher's chdice of activities, organization, and 
behavior, and the consequences for student engagement, classroom 
processes, and i;tudent outcomes are important to identify. Teachers 
(seven English and six math) with two cont/asting classes, an average- 
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ability claas and a low-ability class, were identified in the sample. 
Data analyses consisted of statistical c>^mpar isons of observed behavior 
variables in the two types of classes, along with analyses of the^r ^ 
activity\ structures, based upon the narrative^records from the firsf" 
three weeVs and the rest of the year/ 

r 

'4. Ef fects of high heterogeneity within classes ^ Another important 
context is the influence of variation in students' entering achievement 
on classroom orga^zation! and management* Classes with more diversity of 
student achievement/ability would appear to place greater ^detnands on the 
teacher's plannig^, range of activities, monitoring, and individual con- 
tacts. Data analyses wer6 conducjtfei- on a subset of 20 English jpd. 27 

math classes identified as having ro^an entering CAT in the; 33rd to 66th 

I- 

percentile (exclusion o% low yd hij2^ classes avoided confounding enter- 
ing average achievement with hc^geneity^-heterogene^ty) . Relationships 



between teac'her management behaviors, student bahavx^rrs and outcomes, and 
the amount of within-class variation in sti^dents' entering achievement 
were examined through correlation and>multiple regression techniques. 
The n(<rrative records of extremely heterogeneous classes which appeared 
to be effectively taught^ ( in aerms of achievement gains, stuidei^jf htt\- 
tudes, and classroom maoigemJnt c^^jS^xx^yi^Xi^ analyzed,. Analyses 
focused on the teaching and mpnagem^iPtv^r^begi^ these teachers used to 
cope wi,th a high degree of variaticrr^ ^^iri ^^j^er\{ Bnterinf^ achievement. 



5 . R elationships among teacher pregage variables and roanafl^ement 
charac teristics ^ An extensive\questionnairlLwas 'administered, i^sessinK 
teacher beliefs, concerns, attdtudes, iadCJglparcept ions thought to be 
relevant for their nwnagement I y|||J.e\^/-''^\f^^r whether these teacher 
characteristics were r'elate<J to clas^iri^iTOwcacfSses Mtd outcomes, presage 
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variab lea. , derived from the questionnaire were correlated with classroom 
♦ 

behavior variables and pupil outcome measures ^ebta^ned from each 
teacher 'tJ classes. In addition, the interviews with the teachers in the 
subsarajiles erf more and less effective teiachcrs were content coded, This 
analysis sought to identify differen'fces in planning activities, percep- 
jtions Of management and organization activities and their iraportan/:e, and 
the teacher's decision-^making styles, * . ' 
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Table 1 

Betwcfen-ubaerver Agreeraertt of Component Ratings 
for Single Observations 

4 

Variable 

Nutaber Component Rating Variable p £ 1 

0 

01 .Teacher descrilJ'es obiecrivea clearly .54 .003 

02 Teacher considers attention spans .73 .001 

03 Teacher provides assignmjyiits for 

different students .68 .001 

04 Occurrence of verbal class \ 



participation 



.55 .003 



05 



Teacher usea a -variJty of materials .44 .012 

). 

.35 . .041 



06 Materials are ready and in sufficient 
* quantity 



07 Materials effectively support 
instruction 



.46 .010 



08 Teacher gives clear directions for use 

of materi.>l8 \ •■' .64 ». ' .001 

09 Teacher has distracting ynannerisms • .65 .001 

id Teacher maintains eye contact with 
' students 

11 Teacher's presentation of. materials 

• is clear . 

12 Teacher's presentation is adApted to 

different ability levels .56 ^ .004 

13 Teacher provided and/of seeks I 

rationale and analysis '"^7 .003 

14 Teacher states desired attitudes , .10 .103 

15 High degree of pupil success .^6 .009 

16 Cqntent is related to pupil interest 

and background ' ^67 .001 
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Variable- 
Number 



y Table 1-cont inued • ' 

CoinpGnenr Rating VariaMc ^ ^ 



17 Teacher provides reasonable work 

standards 

18 Amount^ of positive reinforcement 

19 Teacher signals appropriate behavior 

20 Teacher Ireinforces inattentive - 

behavior 

21 Teacher displays consistency in 

dealing wj.th*^behavioxf 

22 Amount of . disruptive behavior 

23 Source of disruptive behavior 

24 ' Teacher stops disruptive behavior 

quickly 

'25 Teacher gives rules ' or procedures to 
stop disruptive behavior 

X 

26 Teacher criticizes orNiustifies 

authority to stop disruptive 
^ behavior ^ 

27 Teacher punishes to stop fiisruptive 

bjmavior 

28 Teacher ignores dis^^i^t ive behavior 

29 Teacher has a conference to stop 

disrupt ive behavior 

30 Teacher displays listenipR skills 

31 Teayhfer expresses Eeelings 

32 Teacher is receptive to student input 

33 Teacher is oriented to student needs 



^ 34 Teacher nurtures student, affd'ctive 
skil Is 

35 Class has tank-oriontt'd focus 



.26 
.38 
.48 

.26 . 

.39 
.22 
0 



0 



0 
0 

0 

'.31 
.30 

' .41 
.23 

.56 

f 

.65 



.104 ^ 

.029 

.007' 

.101 

-024 
.141 



\ 



.081 
.069 
.019, 
.129 

.002 
.001 
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Table Itco'U inued 

Variable * . d < • 
Number Component Rating Variable . J)^ ^ 



36 Teacher' encourages group cohesivenes* 

37 Amount of inappropriate behavior 

f 

38 Teacher stops inappropriate behavior 



behavior 



k\ Teacher punishes to stop 
inappropriate l/ehavior 



• ^ 42 Teacher ignores inappropriate 
behavior . , V 

43 Teacher haa conference to stop 
inappropriate behavior 

I ' 44 Teacher signals desistance of 
inappropriate 'behavior 



.74 .001 
;71 • .001 

.29 .092 



39 Teacher gives rules or ,»roctf<iute8 to ^, 

8top' inappropriate ^behavior . • . ^ ^ 

40 Teacher criticizes or justifies • ] 
authority to, atop inappropriate ^^^^ 

.57 .002 

; .55 ,«p3 
* 

.29 .073 

;o5 .^o» 



ii^r^ Data are from observer pairs in 23 observations. - The 
intraclass correlation estimates the proportion of ^^^^^^^^''^^ 
variance that is reliable. The unreliability of Variables 22 through 29 
appears to be attributable to the low variance of those measures during 
the reliability observations. 



Tablo 2 

Between-observer ARreoment of Stu<:ieiin Engagement 
Rating Cate-gories for Single Observations 



Category 


P 


P < 


Definitely on task, ■ academic 


.71 


.001 ' 


Probably on-task, academic 


0 


- 


Definitely oti cask, procedural 


.67 


.001 


Probably on task, procedural 


.65 


.001 


Off task, sanctioned 


.78 


.001 


Off task, unsanctioned 


.74 


.opi 


Dead time 


0 




On 1 ask, academic 


.71 


.001 


On task, procedural 


^ .68. • 


.001 


On task, total , 


.78 


.oni 

) 1 



Note: Data ar^ from observer pairs in 2 J observations. The 
inlraclass correlation coefficient p, estimates the proportion of 
individual observer variance that is reliable. The iinrdl lability of 
Dpad tinv- appears to hav • been caused by its very low occurrence during 
the reliability observations. \ 
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Table 3 

ftetwoen-p^r Lod Stability of Component Ratings 



J x During the First Three Week« 



Variable • . EnRlish Math . 

Number ^ Variable Description (50 cla88e8) - (52 Cjasses) \ 



instruction, 




01 Teacher describes obiect ives .clearly |^ .60* 

02 Teaclier^considers attention spans /S?* • .59* 

03 Teacher. provides assignments for . ^ 

differeoT^t^dents -4^* t^^* 

OA Occurrence of verbal class - \' 

participation *" 0 .59* 

05 Teacher uses a variety of materials .16 .68* 

Ofr Materials are ready and in sufficient^ 

quantity " - -^9* .34* 

^> ^ 

07 Materials effectively su(A>rt 



9 

• 77^ .67* I. 



08 • Teacher j?ives clear directions for use 

of materials .66* . .62* 



09 Teacher has distr^ict ing mannerisms .82* .38* 

10 Teacher maintains eye contact with 

students i 



7 

.69* .63* \ 



11 Teacher's presentation of materials^ 

is clear -66* .57* 

12 Teacher's presentation is adapted to * 

different ability levels .56* .50* 

13 Teacher provides anH/or seeks 

ratioaale and analysis .58* ,61* 

14 Tt>acher sLnti»s des i red ^at i tudes .77* . /2* 

15 High degree of pupil suv^ces>^-^^^ .49* .55* 

16 . Content is related to pupil interes 

and background -^7* 



. Table 3-continue<i 



Variable . 
Number 



Variable Description 



Engl^ sh Math 
(50 classes) (52 Classes; 



17 

18 
19 
20 

21 

r 

22 
23 
2/* 

25 

26 

27 

28 

29 

30 
31 

\ 

33 
34 



Teacher provides rea senate work 
standards 

Amount of pos it i ve r^infoi^cement 

Teacher signals appropriate behavior 



Teacher neinforces inattentive 
behavior - 

Teacher displays consistency in 
dealing with behavior 

Amount of disruptive behavior 

^ource of disrupti<fe behavior 

Teacher stops disruptive behavi/5r 
quicfkly . ^ 

Teacher j^ives ru les or procedures to 
stop df srupt ive behavior 

Teacher criticizes or justifies 
authority to «top disruptive 
behavior 

Teacher punishes to stop disruptive 
behavior 

Teacher ij^nores disruptive behavior 

Teacher has a conference to stop 
disrupt ive behavior 

Teacher displays listening skills 

Teacher expresnes feelings 

Teacher is receptive to student input 

Teacher is oriented to student needs 

Teacher nurtures stWent affectivfe 
skills 



y> Class has task-oriented focus 



.60* 
.58* 
. jO* 

.73* 

.79* 
.67* 
.56* 

.52* . 
.43* 

.59* 

.60* 
.39* 

.29 

.52j* 
.60* 

■■.4>* 
.48* 

.31 
.49* 



.64* 
.70* 
.43* 

-.15 

.48* . 

.30 

.37* 

.50* 
.12 

.60* 

.85* 
.84* 

00 ' 
.69* 
.65* 
.37* 
• .43* 

.52* 
,(S2* 



Table S-'contimie^ 



Variably EnpUah Math 

.Number Variable Description ^ (50 classes) (52 Classes) 

36 Teacher enioura><es group cohesiveneos .53* .54* 

37 Amount of inappropriate behavior .38 .14 

38 Teacher sdbps iaappropriate behaviO'^ 

quicklyf .^0 

39 Teacher Rives rules or procedures to 

stop inappropriate behavior -H '21 

40 Teacher, criticizes or lustifiee ' 

authority to stop inappropriate ^ 

behavior \ -70* ,22 

41 Teacher punishes to stop 

inappropriate behavior .85* ••28 

42 Teacher ij^nores inappropr iate^ ' 

behavior • .59* .59* 



4*J Teacher has conference to stop 
inappro£r ia'te bohavi or 



00 .03 



44 T(>acher sijcnals des istance of ^ 

inappropriate ^jehavior ,30 ' .Of 



r 



c 



< .05. , . 

Nolo: Coi'f f icients of cur re Li t iort reported in this table aro 
intraclass correlations, which provide an estimate of the proportion of 
total variance that is stable between periods (within teacher). v 
Variables 3^ through 44 wer^> added to the pet of rat ings "after the 
second week of observation, so the stabilities reported for these 
variables may be affected by binall numbers of observations. In a few 
classes, no observations were made on Variables 37-44; 
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Table 4 

The Consiatenc^ Between Periods (Within Teacher) 



of Class Mean Resixiual Ach ievement and Atbdtude Scores 



Math teachers (n " 26) 

Clasb mean residual achievement 
Class tnSan Student Rating 
of Teacher 
English teachers *(n * 25) 

Class mean residual achievement 
Class mean Student Rat.ing 
of Teacher 



PI 



,49/ 

...\ 



.48 



.64 




.66 



.16 



.65 



.76 



£.1 



..01 



.001 



.01 



.001 



■ / 
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Note: Data werl' obtained for 'each teacher in two classes. (p V ) \ 
Intraclfiss correlations estimate tht Vcons istency of each variable vhen ' 
the estimate is based upon one class (p^) or the average of two classes 
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Table 5 

^.Math l^ntercorrelatiiiin Matirices for First Three Weeks 







(N » 52") 




• 
















Variable 


Res idual 


1 ^ 

SRT. 


CAT78 


.\CH 


!• 


Off-task, Unsanctioned 


--11 


.01 


-.38^ 




2. 


On-task, Academic 


.06 


.25 


-.03 


-.02 


3. 


On-task , 


.'19 


.18 


.23 


.27 


4. 


Disruptive behavior 


-.23 


-.00 


-.29 » 


-.34 


5. 


Inappropriate behavior 


-.19 . 

) 


.07 


-.15 


-.20 


6. 


Residual 


.24 V 

• 


-.01 


.27 


7. 


SRT 






-.09 


-.01 


8. • 


CAT78 ' 








.96 



9. ACH • _ 



. — . f 

Math Intercorrelation Matrices for Rest of Year 



(N = 52) 





Var iable 


Residual 


SRT 


CAT78 


ACH 


I . 


Off-task, Unsanctioned ' 




-.01 


-.23 


-.33 


2.. 


On-task, Aca*Semic 


' .28 


.20 


.27 


: . 34 V 


3. 


On-task 


.32 


.15 


\ .31 


.39 


4. 


Disruptive behavior 


-.30 


.05 


-.21 


-^29 


5. 


Inappropriate behavior 


-.30 


.09 


-.17 


-.24 


6. 


Res iduial 




.24 


-.01 ' 


.27 


7. 


SRI 






- .09 


-.01 


H. 


f^T78 










9. 


ACH 

• i 
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Table 5-"continued 
English Intercorrelation Matrices for First Three Weeks 

(N » 50) 



Variable Residual SRT CAT78 ACH 



6. Residual 

7. SRT 

8. CAT78 

9. ACH 



English Intercorrelation Matrices for Rest of Year 

(N - 50) ■ 



9. ACH 



< .05 is indicated by an underlini'. 
£7 .01 in indicated by two underlines 



1. Off-task, Unsanctioned .13 -.22 ~ '?r: 

2. On-tasU, Academic .18 .12 jli ' ^21 

3. /0n-ta8k .02 .21 ^ , ^ 

4. Disruptive behavior -.17 "l?!. 

5. Inappropriate behavior -.13 • "■^3 ' -.26 ".• 



\ 

-.13 .05 ' .29 

-.12 -.14 
.97 





Variable 


• 

Residual 


SRT 


CAT78 


ACH 


r. 


Off-task, Unsanctioned 


-.18 


-.46 


-.10 


-.14 


2. 


On-task, Academic 


.1'4 


-.02 


^32 


.34 


3. 


t)n-ta8k 


.19 


.28 


.25 


.29 


4. 


Disruptive behavior 


-.0«J 


-.40 


. -.23 


-.24 


5. 


Inappropriate behavior 


-.23 


^.43 


-.26 




6. 


' Residual , 




-.13 


.05 


. 29 


7 . 


SRT 






-.12 


■ -.14 


8. 


CAT 7 8 
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TIME LINE FOR 
THE JUNIOR HIGH CLASSROOM ORGANIZATION STUDY 



Spring, 1978 



Mld~Augu8t, 1978 



Contact made with principals of 9 of the 11 

Austin junior Mgh schools. Meetings were 
held with the English and math faculty members 
to dlKcuss the project and secure 9 
volunteers. ,^ 



* 

Two other junior high principals cont-icted an'd 
facul jres met with to obtain volunteers. 



August 21-25, 1978 



Observer .Training at the R&D Center, lasting 
about 25 hours' over a five~<iay period. 



August 28, 1973 



First day of school. Eighteen observers in one 

class each of 53 teachers (25 English and 26 
. math) in 11 junior high schools. 



August 28, 1978- 
•September 15, 1978 ^ 



irat three weeks t)f school. Fifty-three 
teachers, two classes each, seen by 19 
observers for a total of over 575 observation 
hours, or approximately 11 observations per 
teacher. 



Sept€?h>ber 25, 1978- 
. May 1, 1979 



Fifdy-one teachers, two classes each, seen 

approximately every three weeks by one of five 
regular observers for a total of over 850 
observation hours or approximately 17 
'observations per teacher. 



April 9-27, 1979 



Studenta in observed classes given the Student 

Rating of Teacher conn to fill out during ♦the 
last regularly scheduled observation of the 
class . 



\lay 1-18, 1979 



May, 1979 



Students In observed classes given tl r 

\ COET-developed curriculum-based achievement 
V<^8t.8 in Engllphiiand math. 

Teachers were interviewed and they filled out a 
questionnaire concerning organization and 
, management techniques, planning and beliefs 
about the task of toacfilng. 
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C ' ^ NARRATlVr, 

■jciiil;cr *> Schuol « Siihjtvt « P(.M*iod " Ohseivcr " 

Date ^ Students Present Pag^ 



ART: 
































• 














— 















































STOP ; 

erJc 



2. 



3. 



4. 



5. 



6. 



7. 



8. 



9. 



10. 



11. 



12. 



13. 



15. 



16. 



17 



18. 



19. 



20. 



21. 



22. 



23. 



24. 



of 



'•5 5l 



■ / 

Student Engagement Ratings 



teacher # ^ School # Subject # Period # Observe* # 

"^^ate , Number of Students • , 

. ^ ^ 



i ' 2 3^ A 



Time 






4 




Format /Cpde 








pV 

J 


Topic 




— 1 lifc 


<■ 




— — 

# in Room 










9 def. on 9 acad. 










i^rob. on, acad. 










if def . on, l^oc* > 










#'prob, on^ proc. 










# off, sane. 




1 






if off« unsanc. 








; — K 


if dead time 










# can^c see 











erJc 



A6 
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1 |j) U 0 COMPONENT RATINGS 

Teacher # School ff Subject /f Period # Observer # 



Date 



Code 



# of Students Present 



5 4 3 ; 2 1 

5 4 3 2 1 

5/, 4 3 2 1 

5 4 3' 2 1 



"5 4 3 2 1 



5 4 3 2 1 
5 4 3 2 1 



5 4 3 2 1 

5 4 3 2 1 

,5 4 3 .2 '1 

5 4 3 2 1 

5 .4 3 2 1 

5 4 3 2 1 



5 4 3 2 1 
5 4^21 
5 4 3 2 1 

5432)1 



5 4 3 2 1 
5 4 3 2 1 



1. LeBson Design 

a. Describes objectives 
clearly 

b. Attention spans 
considered 

c. Assignments for 
different students 

^ d. Occurrence of verbal 
class participation 

2. Locating. Constructing, 
UsinR Materia ls 

a. baes a variety of 
materials 

b. Materials ready 

c. Materials effectively 
support instructions 

d. Clear directions 

3. Presenting Information 

a. JDistractlng mannerisms 

b. Eye contact 

C. Presentation clear 

d. Adapted to different 
levels 

e. Provides/seeks 

, rationale and analysis 

4. Developlnjai Attitudes 

a. Stiates desired 
attitudes 

b. High degree of pupil 
success 

c. Content related to 
pupil interest/back- 
ground 

d. . Reasonable work 

standards 

5. Managing Pupil Behavior 

a. Amount of positive 
reinforcement 

b. Signals appropriate 
behavior 



5 4 3 2 1 



5 

5 



4 

4 



5 
5 
5 

5 
5 



4 
4 
4 

4 
4 



5 
5 
S 

5 
5 

5 
5 



4 
4 
4 

4 
4 



4 
4 



3 
3- 



3 
3 
3 

3 
3 



3 
3 
3 

3 
3 



3 
3 



2 1 
2 1 



2 
2 
2 

2 
2 



1 
1 
1 

1 

1 



2 
2 
2 

2 
2 



2 
2 



1 

1 

1 
1 

1 
1 



5. Managing P upil nhvr. (Cont. ) 

d. Consistency in dealing , 
with behavior 

e. Amount of disruptive 
* behavior- 

f. Source of disriiptive 
behavior 

^, Methods of Handlin g 
Disruptive B^ehavior 

a. Stops quickly K(i|j^ 

b. Gives rules and procodurcr 

c. Criticizes/justifies 
authority 

d. Punishes 

e. Ignores 

f. Conference 

7. Interacting Effectively 

a. Listening skills 

b. Expresses feelings 

c. Receptive to 
student input 

d. Oriented to student needs 

e. Nurturance of 
affective skills 

8. Classroom Climates 

a. Task-oriented focus 

b. Teacher encourages 
group coheslvenesfl 



5 4 3 2 



1 9. Amount of inaypropriate 
behavi ir 



4 3 



5 
5 
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c. Reinforces inatt< 
behavior 



47 



4 
4 

A 
4 



5 4 

5o 



10» Teacher^ s reaction to 
inappropriate behavior 

2 1 a. Inapproprlcite behavior 
is stopped quickly 

b. Gives rules or procedureB 

c» Crltlcizes/jusylf les 
authority 

d» I?\inlshcs 

Ignores 

^ f. Conference. 

3 2 1 p- Desist 



3 
3 

3 



2 
2 

2 

? 



Teacher ' School I Subject 



Start Stop » Subject/activity 



'X 



I 

i 



. is LOG 



M 



Period # 



Observer # 



Date 



# in class 



^Page of 



Start Stop # %bject/actlvity 



Start Stop #. Sul)ject/Activity 



